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Abstract 

The purpose of this paper is to present cost and 
error data collected during the development cycle of a 
large-scale software effort, to analyze this data in com- 
parison with other available data from similar projects, 
and to evaluate the effectiveness of the techniques utilized 
on the project. The project being reported on is Com- 
puter Sciences Corporation's development of the Central 
Flow-Control Software System for the Federal Aviation 
Administration's Air Traffic Control System Command 
Center. Analysis of the cost data provides insight not 
only into the added development costs associated with 
severely limiting module sizes, but also into the effec- 
tiveness of various cost estimation techniques. The 
error data analysis supports the usefulness of the soft- 
tv-are engineering techniques which were u3ed on the 
project in conjunction 'with definitive module-level test 
requirements. The paper provides a foundation upon 
which to establish the development and, data collection 
environment for future software systems. 


Introduction 

Major software development projects employing con- 
trolled software engineering techniques occur infre- 
quently over the life of an organization. It is even more 
uncommon for these controlled projects to have the man- 
agement support required to collect sufficient data to 
evaluate the techniques utilized. In order that subsequent 
developers may have the opportunity to select affective 
software engineering techniques, more project details 
need to be collected, evaluated, and the results stored 
for their use. 

This paper describes a major software development 
project, the software engineering practices employed, 
the data collection procedures and results obtained, and 
conclusions about the effectiveness of the practices as 
implemented on the project. The experiences gained 
from the project were not of the controUed, psychomet- 
ric variety; the budget and deadlines were real, and the 
various lapses in data and, perhaps, in resolution, re- 
flect these facts. Taking this Into consideration, the 
following material is presented, not as conclusive proof 


of the efficacy of a particular methodology, but as ex- 
periences resulting from a representative large-scale 
development project. 

Project and Approach 

The Federal Aviation Administration (FAA) operates 
an Air Traffic Control System Command Center (ATCSCC) 
whose function is balancing the flow of air traffic so that 
in-flight delays are minimized. The Central Flow Con- 
trol (CFC) System provides automation support for this 
function. A computer complex in Jacksonville, Florida, 
is linked with die major FAA nationwide facilities to pro- 
vide up-to-date information about proposed flights and 
in-flight movements. This demand information is fed 
into a data base along with airport capacity information, 
and is subsequently used by ATCSCC personnel in an 
on-line query environment. The overall system can be 
described as an on-line (inquiry), real-time (flight data), 
transaction-oriented (independent asynchronous activities) 
information system. 

The project was initiated in late 1975. ft was de- 
cided to establish a rigid software architectural require- 
ment and functional (stimulus/ response) specification to 
ensure that the system would be able to evolve along with 
the application. Maintainability with an emphasis on 
modifiability 1 was the prime objective prior to contract 
initiation. However, a more recent grouping of these 
factors indicates that flexibility tended to become the 
primary goal, with maintainability second, and reliabil- 
ity a weak third for the initial system." 

Toward this end, the FAA specified the functional 
requirements,' 1 utilizing an existing hardware configura- 
tion, and provided a baseline operating system. Com- 
puter Sciences Corporation ;CSCi was competitively 
awarded a contract to modify the operating system, 
develop data base management and applications software, 
and provide support software for system development, 
generation, test, and performance evaluation. The 
award was made in April 1977, and the system was de- 
livered in January 1979, six months prior to the opera- 
tional readiness date of July 1979. 

The selection of a development methodology was 
based as much on management criteria as on technical 
criteria. It was decided to move stepwise through the 
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development process, checkpointing each phase by base- 
lining the output. With the requirements definition phase 
and the functional specification phase completed and 
their results baselined, the remaining development effort 
was allocated to three major phases: (1) the system de- 
sign phase; (21 the unit design, code, and test phase; 
and (3) the system test phase. The final two phases were 
performed four separate times. Each time a portion of 
the software (called a Build! was implemented to demon- 
strate a subset of the functional capabilities of the sys- 
tem. 

Software Engineering Methods 

It became clear that successful execution of the' 
development phase (within both budget and schedule) 
would require careful front-end attention paid to: 

(1) product definition, (2) tool utilization, (3) practice 
standardization, and (4) organizational structure. An 
attempt was made (with varying degrees of success) to 
define software and documentation products so that they 
would evolve naturally, and so that as much of these 
products as possible would be in machine- readable form.. 
A set of tools and practices were selected to assist in 
those areas which had been troublesome in the past, most 
notably control and communication. Finally, an organi- 
zation was formed based upon a Work Breakdown Struc- 
ture (WBS), which was to serve as an accounting, data 
collection, and referencing mechanism as well as a basis 
for scheduling. 

The system design phase involved the least infusion 
of modem practices and yielded the least amount of soft- 
ware engineering data. The products of the phase were 
(1) the Program Design Specification (PDS), (2) the Sys- 
tem Development Plan, and (3) the System Test Plan. 

The PDS was developed using Hierarchy plus Input- 
Process -Output (H-CPOl diagrams. The Hierarchy (H-) 
diagrams eventually evolvedinto execution diagrams, 
which gained reasonable support, but the IPO diagrams, 
which met with some initial success, were quickly sup- 
planted by Program Design Language (PDL). The ?DL 
allowed for six basic structured constructs. Data was 
aiso addressed hierarchically in the PDS, and the 
CODASYL Data Description Language was employed."* 
This complemented the PDL, and both were updated for 
inclusion in the final documentation. 

The unit design, code, and test phase was the most 
amenable to incorporation of modern practices and was 
the source of the most useful data. Additionally, the 
organization was adjusted during this phase to provide 
both a Quality Control group and an Independent Data 
Base Administrator. 

Unit design was accomplished using PDL, and the 
unit test specification was generated by automated analy- 
sis of the PDL. This cool aiso verified the PDL for 
compliance with project quality standards. The test 
specifications were for unit testing based upon the 
decision-to-decision (DD) path structure of the design. 


The DD paths were determined from the constructs util- 
ized in the PDL and were a relationship of the number of 
possible branch paths between constructs. The DD paths 
eventually demonstrated some highly advantageous prop- 
erties (discussed subsequently In the section entitled 
"Presentation of Reduced Data”). Units, or modules, 
were constrained to single entry, single exit, and single 
function. They were documented at this stage by a 
machine- readable Prologue, which contained identifica- 
tion, operational-characteristic, cross-reference, data- 
definition, and processing-logic information. ^ Prologues 
were also automatically analyzed for completeness. 
Modules were subjected to Walkthroughs, and the re- 
sultant error data, together with weekly resource ex- 
penditure sheets, the module PDL, Prologue, and Test 
Specifications were incorporated into individual Software 
Engineering Notebooks. 

Unit coding and testing was baaed on the design in the 
PDL and Test Specifications. Structured code (in 
JOVIAL)^ was derived from the PDL, and Test Proce- 
dures were developed based upon the Test Specifications; 
both of these items were then incorporated into the note- 
books. Data interfaces were controlled by use of the 
JOVIAL data-descnption facility, COMPOOL, which was 
regulated by the Data Base Administrator. Resource 
utilization and error data were collected as they were in 
the previous phase; this data is presented and evaluated 
later in the paper. 

The system test phase was carried out for each build 
by an Independent Test Team composed of both developer 
and user personnel. System Test Specifications and 
Procedures, in contrast to those at the module level, 
were for functional testing, and were traceable back to 
the requirements definition phase through the functional 
specifications. Errors were recorded on Test Team 
Trouble Reports, which were included in Build Test Re- 
ports. 

Automated documentation tools were employed at 
the system level. The JOVIAL Automated Verification 
System (JAVS) 3 was modified for this project, and was 
used to produce program hierarchy 'calling tree), pro- 
gram structure (DD paths), and cross-reference infor- 
mation, as well as to support the degree of test case 
coverage obtained. Intermediate JAVS outputs were 
also scanned by a specially developed software tool, 
which produced a Data Item Dictionary. 

CFC Software Data Collection 

Three general categories of data were collected on 
the CFC Project: activity data, software module struc- 
tural data, and software error data. The activity data 
collected was man-hours expended by project personnel 
at the software subsystem level. Software module struc- 
tural data Included counts of the total number of execut- 
able source statements plus a count of the number of 
DD paths from the design for each module. Software 
error data included both walkthrough and execution time 
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errors. Walkthrough errors were recorded by quality 
control at the design walkthrough, and execution errors 
were recorded by the responsible programmers or the 
Independent Test Team, depending on the level of test- 
ing. Error data was collected both at the module (or 
unit) ievel and at the system level. 

The procedures for data collection were based on the 
types of data and the mechanism used to collect each 
type. The data types and the corresponding basic collec- 
tion devices were as follows: 

• . Activity data 

Personnel: time accounting given at the 
subsystem level 

• Module structural data 

Physical structure: module source code 

Logical structure: module PDL and test 
procedures 

• Software error data 

- —Module: programmer error log 

- System: test team trouble reports 

Control of data collection was maintained within the 
development departments and monitored by the quality 
control staff on a continuing basis. Organization and 
delivery of the final data package wa3 performed by the 
quality control staff. The following subsections describe 
in detail the forms and procedures used to collect and 
summarize the three data types. 

Activity Data Collection 

Time data in man-hours was collected for all per- 
sonnel on the project at the functional subsystem level. 
This time data was collected by means of a CFC Project 
Data Collection Form chat was distributed weekly and 
filled out along with weekly time cards by each person on 
the project. The number of man-hours expended in each 
subsystem for design, code, test, library maintenance, 
throwaway code development, management, and quality 
assurance functions performed within the subsystem was 
compiled from this data. 

The data was collected for each of the four builds of 
the system. Within each build, data was presented for 
die Application/Simulation (APS) Subsystem, the Data 
.Assembler (DA) Subsystem, the Data Base (DB) Subsys- 
tem, and the Data Reduction and .Analysis (RA) Subsystem. 
These subsystems represented the major functions per- 
formed by the CFC System that were programmed in 
JOVIAL. The time spent in functional design and system 
level testing was not included in the build man-hour re- 
sults. 


Software Structural Data Collection 

Two types of software structural data were obtained 
from the CFC Development Project. The data was ac- 
quired from examination of the module source code and 
PDL. The module source code data was an estimate of 
the physical size of a module measured by a count of the 
total number of executable source statements. The PDL 
data defined the logical structure of the design of the mod- 
ule, obtained by a count of the total number of DD paths 
in the design. Another measure of logical structure 
utilized was a count of the number of test cases used to 
exercise all the DD paths identified within a module 
during the design phase. The test case data was obtained 
from the test procedures in the Software Engineering 
Notebooks. 

Software Error Collection 

Software error data on the CFC Project was collected 
during the design and testing phases. Errors detected in 
the design phase were measured by the total number of 
design errors discovered during the design walkthrough 
of a given module. These error counts were collected 
by the quality control staff during each walkthrough. 

Errors encountered during module or unit level test- 
ing were collected by the responsible programmer and 
summarized for each build. System level errors were 
collected by the Independent Test Team for each unsuc- 
cessful run. The failure information was derived from 
an analysis of the program output. If a failure was 
caused by more than one error, all errors were listed. 
Errors were also recorded during system acceptance 
testing. While not on a build basis, these errors pro- 
vided Information about problems encountered during the 
integration of the final system. 

Presentation of the Data 

This section presents the raw data collected on the 
CFC Project. Data is presented in the three categories 
described in the preceding section, and is presented for 
every build in which it was available. 

Activity data is shown in Tables 1 through 4. 

Tables 1 through 3 show the man-hours spent per subsys- 
tem in the activity categories of detailed design, code, 
unit test, library maintenance, throwaway code develop- 
ment, and management and technical direction by task 
leaders in the subsystem plus quality control functions 
performed by subsystem personnel. Data is presented 
for Builds 2, 3, and 4 of the system. Build 1 of the sys- 
tem is not presented since it occurred at a rime prior to 
the institution of the reporting mechanism. However, 
total man-hours statistics are av ailab le for Build 1. 

Table 4 presents the total man-hours spent per subsystem 
for each of the four builds of the system. Tables 5 
through 7 show the number of executable lines of code, 
the number of DD paths, and the number of test cases. 
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TABLE 1. BUILD 2 MAN-HOURS IN REPORTED ACTIVITY CATEGORIES 
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TABLE 2. BUILD 3 MAN-HOURS IN REPORTED 
ACTIVITY CATEGORIES 
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TABLE 4 
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TABLE 6. DD PATHS PER SUBSYSTEM PER BUILD 
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TABLE 3. BUILD 4 MAN-HOURS IN REPORTED 
ACTIVITY CATEGORIES 
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TABLE 5. EXECUTABLE LINES OF CODE PER 
SUBSYSTEM PER BUILD 
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TABLE 7. TEST CASES PER SUBSYSTEM PER BUILD 
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•Includes management and technical direction by subsystem leader and quality assurance functions performed by sub- 
system personnel. 
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respectively, for each of the four system builds. Table 3 
gives the error statistics collected for Builds 2 and 3 of 
the system and the total number of software errors de- 
tected during acceptance testing. Euild 1 was completed 
before error reporting mechanisms were in place, and 
Build 4 results were not available. 

TABLE 3. ERROR STATISTICS FOR THE CENTRAL 
FLOW CONTROL SYSTEM 


. 

BUILD 

DESIGN 

WALKTHROUGH 

MODULE- 
- LEVEL 
TESTING 

STSTEM 

LEVEL 

TESTING 

2 

44 

290 

18 

3 

63 

223 

20 

Acceptance 

Testing 

N/A 

N/A 

21 


"APS and DA Subsystems only. 

Presentation of Reduced Data 

The purpose of this section is to analyze the data 
presented in the preceding section. This analysis at- 
tempts to provide quantitative evaluation of the effective- 
ness of the software engineering techniques utilized on 


the CFC Project. Since the project has been completed 
and accepted by the F.AA, this analysis evaluates the 
final results of the project. 

The first analysis performed concerned the relation- 
ship of the sizes of software entities compared to the 
cost of their development. The size of a software entity 
was measured in terms of both the number of lines of 
executable source code and the number of decision paths 
(DD paths) in the design. Cost was always measured in 
man-hours expended. 

Module Level and Build Level were the two software 
entities evaluated. At the module level. Figure 1 pre- 
sents a plot of the number of man -hours expended versus 
the number of lines of executable code for each of a ran- 
domly selected set of 50 modules. Any conclusive trend 
is not at all obvious by analysis of the curve. However, 
since the true comparison of developmental costs is in 
terms of the number of lines of code produced per man- 
hour expended, further evaluation was performed. 

The data, therefore, was evaluated in terms of the 
number of lines of code developed per man-hour (a 
"relative” measure of cost) as a function of the number 
of lines of code in the module (Figure 2 ). If a curve 
could be formed from the data, the optimal module size 
would be the highest point on the curve (i. e. , the module 
size for which the maximum number of lines of code 
would be developed in each man-hour expended). 

The data presented in Figure 2 shows that module 
sizes of between 0 and 40 lines of executable source code 
never (in fact, without exception in this sample) produce 
a productivity of more than one line of code produced per 
man-hour expended. However, for modules of greater 
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FIGURE 1. CORRELATION OF MAN-HOURS TO MODULE SIZES 
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FIGURE 2. CORRELATION OF LINES OF CODE DEVELOPED PER MAN-HOUR COMPARED TO MODULE SIZE 


than 40 lines of executable source code (greater than 100 
without exception), the productivity is generally greater 
than one line of code produced per man-hour expended. 

.Although the data presented only reflects develop- 
ment costs and not operational and maintenance costs, 
the general theory of restricting modules to 50 lines of 
executable source code or less does not appear to be cost 
effective when considering developmental costs only. If 
the "single entry, single exit, single function" rule is 
strictly adhered to, the module size should not be utilized 
as a standard. In contrast, it appears that modules of 
40 lines of code or greater should be encouraged. It is 
important to note that the CFC modules, regardless of 
size, followed the "single entry, 3ingle exit, single func- 
tion" concept of module definition. 

In order to support these conclusions, a measure of 
module complexity, number of DD paths in the design, 
was also compared to developmental costs. The same 
50 modules were utilized. This time, the relative cost 
in number of DD paths generated per man-hour was 
plotted against the complexity in number of DD paths. 
Figure 3 snows the results of this analysis. Again, the 
more complex ±e module, the lower the relative devel- 
opmental costs. 


These cost analyses seem to point out that within the 
"single entry, single exit, single function” concept, the 
larger the module, the lower the per-unit-of-size devel- 
opmental costs. 

The results of this analysis seem to indicate that 
restrictions limiting module size should not be a driving 
factor. Single-function modules of 100 or even 200 lines 
of executable source code should be acceptable. 

The relative size of a build was also analyzed. Fig- 
ure 4 shows the relationship between the size of a build 
in number of lines of executable source code per subsys- 
tem and the number of man-hours expended against that 
subsystem in a build. This plot shows an obvious cor- 
relation of size to cost. With the single exception of the 
DA Subsystem for Build 3, which was accomplished on 
third shift (the implications of which will not be discussed 
here), the cost-to-size relationship is linear. Hence, 
if the single-function module concept is controlled, the 
size of a build appears to have no impact oo the relative 
cost of producing that build. 

Another analysis was performed to evaluate :he re- 
lationship between actual development costs and cost esti- 
mation techniques. The number of man-hours expended 
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per iine of executable source code was compared to the 
number of man-hours expended per DD path in the de- 
sign and to the number of man-hours expended per test 
case ( see Table 9). This comparison was accomplished 
using a coefficient of variation (i. e. , the ratio of the 
standard deviation to the mean). 

TABLE 9. COSTING PARAMETER COMPARISONS 
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The statistical correlation was not diverse enough 
to support differentiation between DD paths in the design 
and lines of code in terms of total cost estimation tech- 
niques. Number of man-hours per test case was shown 
not to be a viable cost estimation technique due to its 
high coefficient of variation (COV). Therefore, a sepa- 
rate analysis was performed to determine a better cost- 
ing parameter that could be utilized at each of the detail 
design, coding, and testing phases of development. 

Since the - known quantity at the completion of each phase 
is DD paths in the detail design phase, lines of code in 
the coding phase, and test cases in che testing phase, 
this information could be utilized to refine original cost 
estimates as a project progresses. The data presented 
on costing provides enough information to support de- 
velopment of initial cost estimation algorithms based on 
actual development products fe.g. , DD paths, lines of 
code, and test cases). 

An analysis was performed on these .ost estimations 
utilizing the data from the APS Subsystem for ail four 
builds. Three cost estimation algori thm s were developed, 
one for each development phase. The basis for these al- 
gorithms is the data previously presented in Table 9. 

The "Lines of Code" algorithm utilizes 1. 34 times the 
number of lines of executable source code to yield the 
estimated number of man-hours. The "DD-Path" algo- 
rithm utilizes 3.34-times the number of DD paths in the 
design to yield the estimated number of man-aours. The 
"Test Case” algorithm utilizes 14.76 times the number 
of test cases to be performed. Ail three algorithms were 
then applied to the other subsystems. These results are 
presented in Table 10. 


The low "average percentage deviation" of the DD- 
path algorithm shows that the number of DD paths pro- 
vides an accurate prediction of the total man-hours 
required. This technique provides the added advantage 
of supporting periodic updating of the estimation as the 
actual number of DD paths is finalized. If PDL is used, 
this variable Is known early (i.e. , at the completion of 
the design phase). 

The effectiveness of the software engineering tech- 
niques utilized was also analyzed. The true effectiveness 
of the software engineering techniques can best be meas- 
ured by the reliability and main tainab ility of the product. 
Although data is not yet available to support definitive 
reliability and maintainability measures of the CPC Sys- 
tem, error rate data was available within CFC and was 
used to estimate the effectiveness of the software engi- 
neering techniques employed on the project. 

In order to evaluate the CFC error rate with some 
defined industry averages, the Rome Air Development 
Center (RADC) Software Reliability Study 9 was utilized. 
This report presents the error rate of two -JOVIAL proj- 
ects at system level testing. This error rate worked out 
to be about 1 error in every 35 lines of code. The CFC 
error rate, calculated from Tables 5 and 9, shows 
1 error in every 28 lines of code, detected at the module 
level. At the system level testing of CFC, an order of 
magnitude fewer number of errors (i.e. , 1 error for 
every 382 lines of code) than at the module level were 
found. During final acceptance level testing, a 3-month 
user/customer-conducted testing phase, still fewer 
errors were found. In the 23, 742 lines of executable 
code discussed in this paper, only 21 software errors 
were detected. This is an error rate of I error in every 
1, 131 lines of code. 

The error rate implies two conclusions about the 
development approach. First, the software engineering 
techniques utilized were very effective. The CFC error 
detection rate comparable to that reported in the RADC 
study was noticed an entire development phase earlier. 
More errors were found earlier, presumably leaving 
fewer errors in the final system. This should result in 
a significant cost savings, since the cost to correct an 
error increases the longer it remains in the code. Sec- 
ond, the testing approach proved to be quite effective. 

The quantification of module level testing requirements, 
specifying that all DD paths in the design must be exer- 
cised at the module level, provided significant advantages 
over the traditional testing approach carried on by the 
programmers. This concept proved to exercise 93 per- 
cent of all the decision paths in the code. Additionally, 
within one subsystem where these unique module level 
test specifications were rigidly applied, the acceptance 
testing error rate was only 1 error detected in ever;/ 

1, 371 lines of executable code; whereas within a subsys- 
tem where module level test specifications were loosely 
applied, the acceptance error rate was 1 error detected 
in every 733 lines of code. This roughly implies an 
overall effectiveness increase of over 100 percent 
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per line of executable source code was compared to the 
number of man-hours expended per DD path in the de- 
sign and :o the number of man-hours expended per test 
case (see Table 91. This comparison was accomplished 
using a coefficient of variation (i.e. , the ratio of the 
standard deviation to the mean) . 

TABLE 9. COSTING PARAMETER COMPARISONS 
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The statistical correlation was not diverse enough 
to support differentiation between DD paths in the design 
and lines of code in terms of total cost estimation tech- 
niques. Number of man-hours per test case was shown 
not to be a viable cost estimation technique due to its 
high coefficient of variation (COV). Therefore, a sepa- 
rate analysis was performed to determine a better cost- 
ing parameter that could be utilized at each of the detail 
design, coding, and testing phases of development. 

Since the known quantity at the completion of each phase 
is DD paths in the detail design phase, lines of code in 
the coding phase, and test cases in the testing phase, 
this information could be utilized to refine original cost 
estimates as a project progresses. The data presented 
on costing provides enough information to support de- 
velopment of initial cost estimation algorithms based on 
actual development products (e.g. , DD paths, lines of 
code, and test cases). 

An analysis was performed on these cost estimations 
utilizing the data from the APS Subsystem for all four 
builds. Three cost estimation algorithms were developed, 
one for each development phase. The basis for these al- 
gorithms is the data previously presented in Table 9. 

The "Lines of Code" algorithm utilizes 1.34 times the 
number of lines of executable source code to yield the 
estimated number of man-hour3. The "DD-Path" algo- 
rithm utilizes 5.34 times the number of DD paths in the 
design to yield the estimated number of man-hours. The 
"Test Case" algorithm utilizes 14.76 times the number 
of test cases to be performed. .All three algorithms were 
then applied to the other subsystems.. These resuits are 
presented in Table 10. 


The low "average percentage deviation" of the DD- 
path algorithm shows that the number of DD paths pro- 
vides an accurate prediction of the total man-hours 
required. This technique provides the added advantage 
of supporting periodic updating of the estimation as the 
actual number of DD paths is finalized. If PDL Is used, 
this variable is known early (i.e. , at the completion of 
the design phase). 

The effectiveness of the software engineering tech- 
niques utilized was al30 analyzed. The true effectiveness 
of the software engineering techniques can best be meas- 
ured by the reliability and maintainability of the product. 
Although data Is not yet available to support definitive 
reliability and. maintainability measures of the CFC Sys- 
tem, error rate data was available within CFC and was 
used to estimate the effectiveness of the software engi- 
neering techniques employed on the project. 

In order to evaluate the CFC error rate with some 
defined industry averages, the Rome Air Development 
Center (RADC) Software Reliability Study 9 was utilized. 
This report presents the error rate of two -JOVIAL proj- 
scts.at system level testing. This error rate worked out 
to be about 1 error in every 35 lines of code. The CFC 
error rate, calculated from Tables 5 and 3, shows 
1 error in every 23 lines of code, detected at the module 
level. At the system level testing of CFC, an order or 
magnitude fewer number of errors (i.e.. 1 error for 
every 3S2 lines of code) than at the module level were 
found. During final acceptance level testing, a 3-month 
user/customer-conducted testing phase, still fewer 
errors were found. In the 23,742 lines of executable 
code discussed in this paper, only 21 software errors 
were detected. This is an error rate of 1 error in every 
1, 131 lines of code. 

The error rate implies two conclusions about the 
development approach. First, the software engineering 
techniques utilized were very effective. The CFC error 
detection rate comparable to that reported in the RADC 
study was noticed an entire development phase earlier. 
More errors were found earlier, presumably leaving 
fewer errors in the final system. This should result in 
a significant cost savings, since the cost to correct an 
error increases the longer it remains in the code. Sec- 
ond, the testing approach proved to be quite efiective. 

The quantification of module level testing requirements, 
specifying that all DD paths in the design must be exer- 
cised at the module level, provided significant advantages 
over the traditional testing approach carried on by the 
programmers. This concept proved to exercise 93 per- 
cent of all the decision paths in the code. Additionally, 
within one subsystem where these unique module level 
test specifications were rigidly applied, the acceptance 
testing error rate was only I error detected in every 
1,371 lines of executable code; whereas within a subsys- 
tem where module level test specifications were loosely 
applied, file acceptance error rate was 1 error detected 
in every 733 lines of code. This roughly implies an 
overall effectiveness increase of over 100 percent 
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TABLE 10. COST ESTIMATION COMPARISONS 



"lines of Cede" "30 ?ath" "Test .Case" 

Algorithm Algorithm Aisorhthm 

Actual Estimated Deviation Estimated Deviation Estimated Deviation 



Man-Hours 

Man-Hours 

( 1 ) 

Man-Hours 
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1122 
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3601 

26.56 

3095 

9.36 

3995 

40.17 

3uiid 2 33 

2030 

1696 

16.45 

1460 

29.08 

1077 

46.95 

3uild 2 DA 

2729 

3047 

11.65 

2354 

13.54 

1993 

26.37 

Build 3 DB 

2381 

3792 

31.52 

2938 

1.98 

2863 

. 62 

3uild 3 DA 

2537 

S397 

112.73 

3l«3 

25.46 

1973 

26.09 

3uiid 3 BA 

3979 

5594 

40.37 

6237 

56.79 

4384 

10.21 

2uild - 33 

2090 

2122 

1.53 

1386 

3.76 

2052 

1.32 

3uild u DA 

-.aa7 

1176 

37.53 

999 

52.36 

384 

73.55 

3uiid - ?A 

2637 

2510 

4.82 

2003 

24.04 

1299 

50.74 

Average 



29.42 


22.51 


30.32 


Deviation 

attributable to using these module level test specifica- 
tions. 

Summary 

In conclusion, the authors feel that a significant base 
line has been established in the quantification of software 
engineering techniques. The success of the CFC project, 
together with the supporting data that was collected, pro- 
vides a foundation upon which to build successor systems. 
The key factors to be considered in setting up -these fu- 
ture system programs is to understand the development 
environment, to collect data during the development ef- 
fort to support the upgrading of projections, and :o sup- 
port the periodic evaluation of the data to provide insight 
into the product. This should provide sufficient visibil- 
ity to keep a project out of trouble, while at the 3ame 
time supporting evolution of more effective project plans. 
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DATA ANALYSIS AREAS AND 

RESULTS 

• MODULE SIZE ANALYSIS 

• BUILD SIZE ANALYSIS 

• COSTING METHODOLOGY RESULTS 

• TESTING METHODOLOGY RESULTS 
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DATA COLLECTED 


ACTIVITY DATA 

- PERSONNEL: TIME ACCOUNTING GIVEN AT THE 
SUBSYSTEM LEVEL 


MODULE STRUCTURE DATA 

- PHYSICAL STRUCTURE: MODULE SOURCE CODE 

- LOGICAL STRUCTURE: MODULE PDL AND TEST 
SPECIFICATIONS 


SOFTWARE ERROR DATA 

- MODULE: PROGRAMMER ERROR LOG 

- SYSTEM: TEST TEAM TROUBLE REPORTS 



DATA PRESENTED 


DATA PRESENTED ON THE MAJOR 
SUBSYSTEMS OF CFC THAT WERE 
CODED IN JOVIAL 

38034 MAN-HOURS OF DIRECT LABOR 
AND 23742 LINES OF EXECUTABLE CODE 
REPRESENTED OVERALL 

OVERALL DATA PRESENTED: 

MAN-HOURS, EXECUTABLE LINES OF 
CODE, DD-PATHS, TEST CASES, 

ERRORS 
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OVER LIMITING SIZE 
NOT COST EFFECTIVE 


LINES OF 
CODE PER 
MAN-HOUR 
EXPENDED 



NUMBER OF LINES OF CODE 


LINES OF 
CODE PER 
MAN-HOUR 
EXPENDED 



MODULE SIZE 


BUILD SIZE 



COSTING METHODOLOGY 


MAN-HOURS PER LINE OF CODE/DD PATH/TEST CASE WERE CAL- 
CULATED FOR EACH SUBSYSTEM FOR EACH BUILD 


RESULTS: 

COV (LINE OF CODE) = 35% 

COV (DD PATH) = 32% 

COV (TEST CASE) = 76% (ELIMINATED AS USEFUL ESTIMATOR) 


APS SUBSYSTEM USE AS ESTIMATOR FOR OTHER SUBSYSTEMS AND 
RESULTS COMPARED WITH ACTUALS 


RESULTS: 

AVERAGE DEVIATION (LINES OF CODE) = 24% 

(DD PATHS) = 22% 


CONCLUSION: 

DD PATHS CAN BE USEFUL IN REFINING INITIAL COST ESTIMATES 



TESTING METHODOLOGY 


• DISCIPLINED APPROACH 

• TEST SPECS GENERATED FROM PDL 

• TESTS EXERCISE ALL DD PATHS IN DESIGN 

• VERIFIED AT DESIGN WALKTHROUGHS 

RESULTS 

• 1 ERROR PER 28 LINES OF CODE AT MODULE LEVEL 

• 1 ERROR 382 LINES OF CODE AT SYSTEM LEVEL 

• 1 ERROR PER 1131 LINES OF CODE AT ACCEPTANCE 
LEVEL 

NOTE: SUBSYSTEM RIGIDLY FOLLOWING PDL/TESTING 
STANDARDS 

1 ERROR PER 1871 LINES OF CODE DETECTION 
AND CORRECTION TIME AVERAGED 8 PERSON 
HOURS. 

SUBSYSTEM LOOSELY FOLLOWING DL/TESTING 
STANDARDS 

1 ERROR PER 733 LINES OF CODE DETECTION AND 
CORRECTION TIME AVERAGED 40 PERSON 

HOURS . BEL-9- 79 
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SUMMARY 


• BUILD SIZE HAS LITTLE COST IMPACT 

• OVER RESTRICTING MODULE SIZE IS 
NOT COST EFFECTIVE IN THE INITIAL 
DEVELOPMENT 

• RIGID ADHERENCE TO THE 
METHODOLOGY CAN REDUCE COST 

• THE METHODOLOGY DID NOT 
SIGNIFICANTLY REDUCE THE NUMBER 
OF ERRORS BUT DID ALLOW EARLY 
DETECTION OF MOST ERRORS 


• THE NUMBER OF DO PATHS IS DIRECT- 
LY RELATED TO LINES OF EXECUTABLE 
CODE AND CAN BE USED TO REFINE 
COST ESTIMATES AFTER THE DESIGN 
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